Support for free-threaded build of CPython #504
Conversation
|
@nascheme I have just pushed a new faber release (0.5.10) including your contributions. Let's see whether they help with this. |
|
Thanks for merging the faber changes. My latest push adds the nogil option for BOOST_PYTHON_MODULE_INIT. It could be simplified if we now require C+11 since there is quite a bit of code to support C+98. The code for this follows closely how pybind11 implements the corresponding option. We might follow what's done by pyo3 in regard to this option and make the default to be true. I've tested with a free-threaded Python build with thread-sanitizer (TSAN) enabled. Running the pytest unit tests using pytest-run-parallel generates no TSAN warnings. That definitely does not mean we are safe from data races but it gives me some re-assurance we have the right locks in place. |
|
For the
The |
|
Thanks for all the context. The PR is still marked as "cannot be rebased due to conflicts". Can you retry rebasing it manually ? (In general, I prefer to rebase a feature branch repeatedly over merging the target branch into the feature branch, as the former allows to fast-forward merge at the end, while the latter doesn't.) |
|
Okay. Are you planning to fast-forward merge without squashing? If so, I'll rebase and also cleanup the commits into more logical ones. If you don't squash them then they could be improved. |
|
I don't know yet. I need to understand the changes better, and, if the commits are self-contained and complete, it might actually be better to keep them separate to make it easier to understand (and possibly debug) them. But at the very least, we need to make them conflict-free to be able to apply and test them. |
For the free-threaded build, it is not safe use borrowed references. Another thread could deallocate the object and cause the reference to become invalid. Replace API calls that borrow references with strong reference APIs.
For the free-threaded build (and possibly the debug build), it is not safe to call Py_DECREF() if there is no valid Python thread-state.
Add pymutex.hpp which implements a re-entrant mutex on top of Python's PyMutex. Add BOOST_PYTHON_LOCK_STATE() macro that uses RAII to lock mutable global state as required.
This indicates that the free-threaded build of Python can keep the GIL disabled when the module is loaded. Without this module flag, importing the module will cause the GIL to be re-enabled. A warning is emitted if this happens.
Implement optional arguments for BOOST_PYTHON_MODULE_INIT and allow the
boost::python::mod_gil_not_used() option. This sets the
Py_MOD_GIL_NOT_USED flag for the extension module.
To define a module that supports free-threaded Python, define it
like this:
BOOST_PYTHON_MODULE(my_module, boost::python::mod_gil_not_used())
{
...
}
2 participants
Summary of changes:
~object_base(). CallingPy_DECREF()with no thread-state is not allowed.Building this requires a fix to faber. I found building Boost.Python very tricky (likely because I'm not familiar with b2 and faber). I'll share my build scripts in case they help someone else (need to tidy them first).
There is another branch that includes doctests converted to use pytest. I only did a subset of the tests (using Claude to assist in conversion). The goal is to use pytest-run-parallel and try to trigger data races or other issues.